Uncertainty and Automation

There are many issues associated with the role of humans in automated systems. Among those that have received the least attention are the questions associated with uncertainty. Uncertainty is inherent in military operations, yet it is often ignored. By making assumptions, explicitly or implicitly, about the environment in which it works, automation can modify evidence of uncertainty, conceal the sources of variability, or even induce uncertainty in the user by neglecting environmental variability. This paper will explore the range of issues and suggest some ways to mitigate the effects of uncertainty associated with automation. ASSUMPTIONS To begin, let us consider the assumptions that underlie this paper. The reader may not agree with them. However, they are the firmly-held opinions of the author and, as such, must be made explicit for the reader to comprehend the argument herein. The first, and most important assumption is that automation is intended to assist the human, not the other way around. It is the human who is ultimately responsible, and therefore, the human who must be in control. The automation and the human must work together to accomplish the human’s goals. It follows from this assumption that in order for automation to do its part of the job, it must communicate with the user. Second, machines are literal-minded. We have often yelled at our machines to “Do what I mean, not what I said!!!” but they can only do what we tell them to do. Humans are good at understanding intention and context. Machines are not. For example, the machine – world interface is limited to what a pre-determined subset of possible inputs. The automation might have privileged access to information that the human cannot access alone. While machines may have privileged access, they can only interpret those inputs in a pre-planned way. Surprise can not easily be accommodated. Third, users have specific expectations about how machines will work and respond accordingly. We expect machines to work consistently, the same way every time the situation is the same. When automation works in unpredictable ways humans do know how to interpret the behavior and may not trust the information. We expect other people to differ in their behavior from time to time, but not automation. We might discount the poor performance of a co-worker who has a bad cold. Sniffling and sneezing are cues to the condition. RTO-MP-088 Paper presented at the RTO HFM Symposium on “The Role of Humans in Intelligent and Automated Systems”, held in Warsaw, Poland, 7-9 October 2002, and published in RTO-MP-088.